source_field_gen
1. Description
This program processes .cf and .txt input files, stamps the source and field names in the output.
2. Input
2.1 Mandatory Parameters
| # | Parameters | Description | Example |
|---|---|---|---|
| 1 | process_config | Path to the process configuration file | /path/to/process_config.json |
| 2 | as_on_date | The date for which the program should run | 2025-01-01 |
| 3 | log_file_path | Path to write log file | /path/to/log_file.txt |
| 4 | diagnostics_file_path | Path to write diagnostics logs for performance and error tracking | /path/to/diagnostics_file.txt |
| 5 | connection-string | Connection credentials to connect to the database | Oracle (Username|Password|IP:PORT/Servicename) and Mssql (Server|Database|UserName|Password) |
2.2 Non-Mandatory Parameters
| # | Parameters | Description | Default Value | Example |
|---|---|---|---|---|
| 1 | log_level | Level of diagnostics written to the log file | info | error/warn/info/debug/trace/none |
| 2 | is_perf_diagnostics_enabled | Flag to decide if performance diagnostics will be logged | false | true or false |
| 3 | delimiter | Separator of columns in input file | " | " |
| 4 | option | Pipe-separated key-value pairs | " " | key1,value1 |
| 5 | skip_if_input_not_present | Flag to decide whether to skip the operation if the input file is not present | false | true or false |
2.3 Process config file 🔗
This file contains:
| # | File | Description | Example |
|---|---|---|---|
| 1 | input_metadata | Path to input_metadata file. | test-bed/M_PC001.json |
| 2 | additional_input | Path to additonal files required by the program. | test-bed/M_IN003.json. |
| 3 | output_metadata | Path to output_metadata file. | test-bed/M_OP001.json |
3.Output
The source and field names are stamped according to the required fields separated by "|". It is of the format:
source|field_name|value|Y|admin|as_on_date|admin|as_on_date
It is loaded into the database into "SourceDataFieldValues" table.